Clinical Data Journey (Phase 4): Using metacore Metadata with metatools for SDTM Dataset Development

Objective

In the previous article, we created a metacore metadata object from a CDISC SDTM specification workbook.

That object now contains standardized metadata describing SDTM datasets, variables, controlled terminology, and derivation information.

But, creating a metadata object is only the beginning.

During real-world SDTM programming, metadata is used throughout the development process to understand which variables belong in a domain, their expected attributes, and whether the final dataset aligns with the study specification.

In this article, we will explore how metacore and metatools can be used to retrieve and validate metadata during SDTM dataset development.

Load Required Packages

library(tidyverse)
library(metacore)
library(metatools)
library(lubridate)

Import raw data

raw_dm <- read_csv("RAW_Demo.csv",show_col_types = FALSE)

Review the raw data:

head(raw_dm)

# A tibble: 6 × 15
  STUDYID   SITEID  Patient_ID   Age Gender RACE   ETHNICITY HEIGHT_CM WEIGHT_KG
  <chr>     <chr>   <chr>      <dbl> <chr>  <chr>  <chr>         <dbl>     <dbl>
1 ONCO-2026 SITE004 SUBJ0001      62 Male   WHITE  NOT HISP…      167.      78.2
2 ONCO-2026 SITE014 SUBJ0002      84 Female BLACK… NOT HISP…      164.     100. 
3 ONCO-2026 SITE010 SUBJ0003      71 Female WHITE  NOT HISP…      168       79.6
4 ONCO-2026 SITE007 SUBJ0004      49 Female ASIAN  NOT HISP…      174       79.8
5 ONCO-2026 SITE002 SUBJ0005      45 Female ASIAN  NOT HISP…      154.      73.2
6 ONCO-2026 SITE002 SUBJ0006      64 Female BLACK… NOT HISP…      145       92.4
# ℹ 6 more variables: BMI <dbl>, BIRTH_DATE <chr>, COUNTRY <chr>,
#   RAND_DATE <chr>, ARM <chr>, TREATMENT_START_DATE <chr>

The raw dataset contains study specific variables collected from the clinical database. These variables need to be transformed into the standardized SDTM DM structure.

Load the SDTM Metadata Object

Using the SDTM specification introduced in Phase 3, we recreate the metacore object so it can be queried throughout the SDTM development workflow.

spec_file <- system.file(
  "extdata",
  "SDTM_spec_CDISC_pilot.xlsx",
  package = "metacore"
)


meta <- spec_to_metacore(spec_file)

The metacore object is now available for querying dataset and variable metadata.

Select the Required SDTM Domain

Instead of manually maintaining DM specifications separately, metadata can be queried directly from the metacore object.

For this example, we will retrieve the DM domain metadata.

dm_spec <- select_dataset(
  meta,
  "DM"
)

The extracted object now contains metadata specific to the DM domain.

Explore DM Variable Metadata

Once the DM metadata is extracted, variable attributes can be reviewed before dataset programming.

dm_spec$var_spec

# A tibble: 25 × 6
   variable length label                              type     format common
   <chr>     <int> <chr>                              <chr>    <chr>  <lgl> 
 1 STUDYID      12 Study Identifier                   text     <NA>   TRUE  
 2 DOMAIN        2 Domain Abbreviation                text     <NA>   FALSE 
 3 USUBJID      11 Unique Subject Identifier          text     <NA>   FALSE 
 4 SUBJID        4 Subject Identifier for the Study   text     <NA>   FALSE 
 5 RFSTDTC      10 Subject Reference Start Date/Time  date     <NA>   FALSE 
 6 RFENDTC      10 Subject Reference End Date/Time    date     <NA>   FALSE 
 7 RFXSTDTC     20 Date/Time of First Study Treatment datetime <NA>   FALSE 
 8 RFXENDTC     20 Date/Time of Last Study Treatment  datetime <NA>   FALSE 
 9 RFICDTC      20 Date/Time of Informed Consent      datetime <NA>   FALSE 
10 RFPENDTC     20 Date/Time of End of Participation  datetime <NA>   FALSE 
# ℹ 15 more rows

The variable metadata includes:

Variable name
Label
Data type
Length
Format

Instead of manually documenting these attributes inside programming code, they are stored centrally within the metadata object.

The programmer can now use metadata as a reference while deriving the SDTM dataset, rather than manually referring to specification documents.

With the required metadata available, we can begin deriving the SDTM dataset while using the metadata as a reference for the expected dataset structure.

Derive Study-Specific SDTM Variables

Metadata describes the target SDTM structure.

The derivation logic, however, remains study-specific and must still be programmed.

For example:

DM <- raw_dm %>%

mutate(

DOMAIN = "DM",

USUBJID = paste(
STUDYID,
SITEID,
Patient_ID,
sep="-"
),

SUBJID = Patient_ID,

SEX = case_when(
Gender=="Male" ~ "M",
Gender=="Female" ~ "F",
TRUE ~ NA_character_
),

AGE = Age,

ETHNIC = ETHNICITY,

ARMCD = case_when(
ARM=="ARM A" ~ "A",
ARM=="ARM B" ~ "B",
TRUE ~ NA_character_
),

BRTHDTC =
format(
dmy(BIRTH_DATE),
"%Y-%m-%d"
)

)

The SDTM dataset is derived using study-specific programming logic, while the metadata provides the standard that the final dataset should conform to.

Retrieve Expected Variables from Metadata

The metadata object defines the variables expected in the DM domain.

Rather than manually creating this list, we can retrieve it directly from the metadata.

Metadata can also be used to maintain SDTM variable ordering according to the specification.

expected_vars <- dm_spec$ds_vars$variable
expected_vars

 [1] "STUDYID"  "DOMAIN"   "USUBJID"  "SUBJID"   "RFSTDTC"  "RFENDTC" 
 [7] "RFXSTDTC" "RFXENDTC" "RFICDTC"  "RFPENDTC" "DTHDTC"   "DTHFL"   
[13] "SITEID"   "AGE"      "AGEU"     "SEX"      "RACE"     "ETHNIC"  
[19] "ARMCD"    "ARM"      "ACTARMCD" "ACTARM"   "COUNTRY"  "DMDTC"   
[25] "DMDY"    
attr(,"label")
[1] "Variable Name"

metadata_order <- expected_vars

DM <- DM %>%
select(any_of(metadata_order))

Retrieving variable names directly from metadata helps reduce manual maintenance and helps keep the programming aligned with the SDTM specification as the metadata evolves.

Compare Metadata with the Programmed Dataset

Missing Variables

setdiff(
  expected_vars,
  names(DM)
)

 [1] "RFSTDTC"  "RFENDTC"  "RFXSTDTC" "RFXENDTC" "RFICDTC"  "RFPENDTC"
 [7] "DTHDTC"   "DTHFL"    "AGEU"     "ACTARMCD" "ACTARM"   "DMDTC"   
[13] "DMDY"

These variables are defined in the SDTM specification but have not yet been derived in this example. Depending on the study and available source data, some variables may be intentionally left unpopulated or derived in later programming steps.

Unexpected Variables

setdiff(
  names(DM),
  expected_vars
)

character(0)

No unexpected variables are present because the dataset structure has been aligned with the metadata-defined variable list.

Review Dataset Keys

Metadata may also define dataset keys, which identify the variables that uniquely identify each record within an SDTM domain.

get_keys(
  dm_spec,
  "DM"
)

# A tibble: 0 × 2
# ℹ 2 variables: variable <chr>, key_seq <int>

In this example, the metadata specification does not define any key variables for the DM domain, so get_keys() returns an empty tibble.

If key variables were defined in the metadata, this function would return them in the order specified. These keys can then be used to help verify dataset uniqueness and support data quality checks during SDTM development.

Check for Duplicate Subjects

DM %>%
count(USUBJID) %>%
filter(n > 1)

# A tibble: 0 × 2
# ℹ 2 variables: USUBJID <chr>, n <int>

Key Takeaways

In this phase we learned how metadata can support SDTM programming by allowing us to:

Retrieve dataset metadata from a metacore object
Explore variable-level metadata
Compare programmed datasets against the SDTM specification
Identify missing and unexpected variables
Use metadata to support SDTM validation

At this stage, metadata is supporting SDTM development by serving as both a programming reference and a validation layer.

In the next phase, we will explore how metadata can be integrated more deeply into SDTM dataset development using sdtm.oak, metacore, and metatools workflows.